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ABSTRACT 

This paper describes teacher usage of the 
microcomputer urograms Test Analysis Package (TAP) and Student 
Problem Package (SPP) to analyze students' test item responses. These 
methods of organizing, analysing, and reporting test results have 
proven useful to classroom teachars. The TAP consists of four 
integrated microcomputer programs to edit, score, summarize, and 
analyze student test data at either the item or objective level. The 
SPP is a program for analyzing student responses on tests baoed on 
the Student-Problem Curve Theory developed by T. Sato. Standard 
reports include student summaries, item summaries, and test 
summaries. Information available for item analysis includes item 
ranking in order of difficulty, frequency distributions by qjintiles, 
point-biserial correlations between response options and total test 
score, and modified caution indices for items (a measure of the 
irregularity of the responses to the items) . The reports also include 
information useful for analysis of student achievement, including 
individual student percent correct data, student ranking within the 
classroom, and objective mastery data. Much of this information is 
organized into a Student-Problem Chart — a useful, easy to use summary 
chart of student responses. Illustrations and examples of these 
reports and their use by classroom teachers are appended. (TJri) 
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Abstract 



The paper describes teacher usage of the microcomputer programs Test 
Analysis Package (TAP) and Student Problem Package (SPP) to analyze students 1 
test item responses. Standard reports include student summaries, item 
summaries, and test summaries. Information available for item analysis 
includes item ranking in order of difficulty, frequency distributions by 
quintiles, point-biserial correlations between response options and total test 
score, and modified caution indices for items (a measure of the irregularity 
of the responses to the items). The reports aloO include information useful 
for analysis of student achievement such as individual student percent 
correct, student ranking within the classroom, objective mastery, and mocified 
caution indices for students (an index of unusual response patterns). Much of 
this information is organized into a Student -Problem Chart, a useful and easy 
to vse summary chart of the student responses, illustrations and examples of 
these reports and their use by classroom teachers rill be provided. 



Mathematicians, computer scie tists, and educators are showing 
considerable interest in the study of item response patterns. Diagnosis of 
response patterns should be an important part of teaching. The analysis of 
response patterns is motivated by at least two factors. First, the belief 
that additional instructional information i„ contained in the analysis of the 
errors students make in responding to test items. Secondly, the analysis of 
response patterns can be useful in test item analysis, where items are 
examined to see if they are functioning properly to aid in building future 
tests. 

This paper will present methods of organizing, analyzing, and report Lng 
test results that have proven useful to the classroom teachers. Teachers 
spend a large portion of class time on testing. Porr-Bremme and Herman (1986) 
surveyed tenth grade teachers and found that 12% of class time is used for 
testing. For each hour a student spent taking tests, these teachers reported 
spending two or more hours in test preparation, scoring, and recording . All 
in all, testing is a time-consuming operation. 

It is only logical that the results of all this testing should be used 
to the maximum benefit of both students and teachers. But too often the only 
information a teacher receives is the final score. Typically, even on 
teacher-made tests, due to the time and effort necessary to get diagnostic 
material from a test, a total correct score is the only information used. The 
Student -Problem Package (Harnisch & Romy, 1985) and the Test Analysis Package 
(Harnisch, Horwitz & Wang, 1984) are two microcomputer programs which analyze 
test data very efficiently and provide much additional information for teacher 
use . 
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Test Analysis Package 
The Test Analysis Package (TAP) consists of four integrated 
microcomputer programs to edit, score, summarize and analyze student test data 
at either the item or objective level. TAP includes a report generator which 
summarizes the descriptive statistics for students, items, and the test, 
producing a frequency distribution of test scores, and optional graphical 
displays on test quality and student performance. TAP also allows the user to 
select a subset of the total test items for inclusion in the item analyses, 
thus helping the user evaluate the best possible test which may be created 
using those items. 

TAP requires data files to consist of one vector (line) of information 
per student (see Appendix 1A) . The first columns consist of the student ID 
information (which can include school and classroom codes), followed by the 
actual item responses. TAP also requires a data file consisting of the answer 
key unless the responses have already been scored as 0/1 (see Appendix 1A) . 
TAP will allow the typing in ana storing of an answer key if this has not 
already been done. 

TAP output includes a student summary, item summary, test summary and 
frequency distribution of test scores (see Appendix 2). There are several 
optional reports. One provides a listing of the student IDs, scores, and 
percent corrects. Another lists the student IDs and the scored item responses 
(0/1) for all items (see Appendix IB). It is also possible to request a 
report of the point-biserial correlations between the response options 
(distractors) and the total test scores (see Appendix 3). 

Another set of optional reports analyzes the student data by quintiles. 
Here the students are divided into five approximately equal groups: the one- 
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fifth scoring the lowest on the total test score, up to the one -fifth scoring 
the highest. The user then has the option of choosing several reports on 
these five groups. TAP will report the score ranges within each group and the 
percent of correct responses to each item by quintile (see Appendix 4). The 
user can choose to have these percent corrects by quintile reported in 
graphical form (see Appendix 5). There is also an option that reports the 
frequency of responses to each item response option by quintile (see Appendix 
6). 

Should the user wish to perform an objective analysis of the student 
data, TAP will accept input of up to 10 objectives (with labels), which items 
map i^to each objective, and the. number of items needed to master each 
objective. The TAP objective analysis report includes a listing of student 
IDs with the percent of objectives mastered 2nd an indication of which 
objectives were mastered (see Appendix 7). This report also includes 
descriptive statistics on each objective, means, standard deviations, standard 
errors, number of students mastering, average item difficulty, reliability, 
etc., (see Appendix 7b), and a graph of the percentage of students mastering 
each objective. Finally the objective analysis includes the frequency 
distribution of objective mastery by quintiles, and the percent of students 
mastering each objective by quintiles. 

Student-Problem Package (SPP) 
The Student -Problem Package (SPP) is a progrera for analyzing student 
responses on tests based on Student -Problem Curve Theory developed by Takahiro 
Sato at NEC Corporation, Tokyo (Sato, 1980; Sato & Kurata, 1977). S-P curve 
theory allows the recovery of additional information from the pattern of 
responses by students to Items. Like TAP, SPP uses a data file consisting of 
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one line per student containing the student ID and item responses. SPP also 
requires a data file consisting of the answer key, unless the responses have 
already been scored (0/1). (See Appendix 1A for examples). 
S-P Chart 

An S-P Chart consists of an ordering of the student item responses in 
the form of a table (see Appendix 8). The problems are ordered from easiest 
(most number of students answering correctly) to hardest, and the students are 
ordered from highest total score to lowest total score. Each row of this 
matrix contains the responses of a student. A »+" indicates a correct 
response while a number represents the distractor chosen. [The »S" and »P» 
notations are discussed in the next section.] The columns of the matrix 
correspond to items on the test. The sum of each row represents the raw score 
(total score) for erch student. For each student the S-P Chart licts the raw 
score (total items correct), the percent of items answered correctly, the 
Modified Caution Index (MCI) and Modified Caution Sign (MCS). [The MCI and 
MCS will be discussed later in this paper.] 

In addition to the S-P Chart, the program reports item statistics: 
number of students getting the item correct, the percent correct, MCIs and 
MCSs for items, and the average MCI (see Appendix 9). Reported test 
statistics inclvde the mean total score, test reliability (Cronbach's Alpha;, 
and the disparity coefficient (ar. index of the classroom/test fit). A short 
student summary includes the average raw test score (and its standard 
deviation), and the average student MCI (see Appendix 9). 

Appendix 10 illustrates further output from SPP in chart form. The 
teacher can quickly see how many students were classified as high and low 
(total test score), and with high and low MJI's (chart 1). Also shown is the 
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set of cut-off values chosen. The number of items classified as hard and 
easy, and with high and low MCI's is shown in Chart 2. 

Through the optional Categorized S-P Chart, the SPP program allows for 
analysis by objective (see Appendix 11) The items, grouped by objective, are 
ranked left to right from easiest to hardest. This ranking is done using the 
average percent of the class that got items correct in that objective. Then 
the items within each group are ordered from easiest to hardest. Otherwise 
the format of the Categorized S-P chart is similar to that of the regular 
chart. 
S-P Curves 

There are two lines that can be drawn on the S-P Chart. The S-curve 
provides a visual display of the proficiency level of the students in the 
classroom (see Appendix 8). Each S is placed at the position in each row 
representing the number of items that student got correct. A vertical S-curve 
represents a homogeneous classroom (one in which most of the students are 
performing very similarly), where a diagonal S-curve is evidence of a 
heterogeneous classroom (a wide range of performance). An S-curve shifted to' 
the right shows a high proficiency level in the classroom. 

A P is placed in each column at that position corresponding to the 
number of students correctly answering the item. In an ideal classroom with 
an ideal test (where everything tested is taught and learned) the S and P- 
curves would coincide. It is quite common to have the S and P-curves diverge 
a little from each other, due to individual differences commonly observed in 
the classroom. If the test is measuring information not covered in the 
classroom, the divergence will be increased. A large divergence (or 
disparity) between the S and P-curves signals a possible mismatch of test 
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objectives and instructional objectives in the classroom. A disparity index 
is given in the SPP analyses for each S-P Chart. A disparity greater than .6 
should be interpreted as a danger signal (Harnisch, 1983). 
Modifi ed Caution TnHpv 

The degree to which a response pattern might be considered unusual is 
represented by a value called the Modified Caution Index (MCI). This index is 
based on Student-Problem curve theory. The MCI for a student is computed from 
a formula that includes such values as the student's total score and the ' 
number of persons that got each item correct (see Harnisch and Linn, 1981, for 
more details). The MCI for an item is computed in a similar manner. The MCI 
ranges from 0 to 1 where a value of 1 represents a response pattern totally 
inconsistent with the expected pattern based on the classroom performance, 
while a value of 0 represents a pattern consistent with the expected 
performance. As a result, the larger the MCI values, the greater the 
indication that this response pattern represents a major departure from an 
expected pattern. Typically, „ M CI greater than .30 means the student's 
total score should be considered with caution. 

The Modified Caution Sign (MCS) is an aid for quickly picking out those 
students with unusual response patterns. An 'A' sxgnifies an adequate total 
score (this cut-off is determined by the teacher) and a consistent response 
pattern (MCI < .30). A 'B' signifies an adequate total score but an 
inconsistent pattern (MCI > .30). A 'C indicates a 1, ; total score but a 
consistent pattern, while a 'D' indicate, a low total score and an 
inconsistent pattern. This way , teacher can quickly read down the MCS column 
and pick out those students with B's and D's. These students' total scores 
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should be considered with caution, and their responses should be examined more 
carefully. 

Modified Cau'cion Signs for items are assigned in a similar manner. An 
item that is relatively hard and has a high MCI will have an MCS of 'X' . A 
hard item with a low MCI will be assigned a 'W , an easy item with a low MCI 
will be assigned a'Y' and an easy item with a high MCI will be arsigned a 'Z\ 
The Data Source 

The data used in the reports found in these appendices came from a 
single classroom of 20 students in a suburban Chicago school district. The 
test was a Basic Algebra semester exam. From this test, twenty- two items have 
been chosen to be analyzed, with those items grouped into four objectives or 
content domains. 

Individual Student Analysis 
TAP and SPP reports contain the basic test information needed by every 
teacher, including a list of student IDs with corresponding total scores and 
percent corrects. If the teacher desires to know exactly which items were 
missed by each student, an optional report will list a 0/1 matrix of students 
and responses (Appendix IB). More detailed information of this sort can be 
found on the S-P Chart (Appendix 8) where SPP provides the actual discractor 
chosen. The S-P Chart also ranks the students for easy identification of 
those students with lowest scores. 

From the S-P Charts, response patterns of students can be examined to 
determine the accural of the individual's total score. Response patterns 
for students are unusual to the extent that students answer some easy items 
incorrectly while answering some difficult items correctly. A wide variety of 
factors may lead to an unusual response pattern. These include sporadic study 
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habits, absence at the time a topic was covered in class, carelessness, a 
common misunderstanding, test anxiety, net feeling well, poor test taking 
skills, a short attention span, or an underdeveloped cognitive structure. 
Also, hidden strengths due to experiences outside the classroom or unusually 
high interest in a specific topic can contribute to unusual patterns of right 
and wrong answers . 

On the sample S-P Chart i„ Appendix 8 it is possible to pick out the 
students with troublesome response patterns by looking for MCIs greater than 
•30 (or an MCS of B or D) . Only one student, the last, has been identified in 
this manner. The MCI of .43 indicates some concern about the representa- 
tiveness of the low test score of 22.7*. This student's test should be 
considered for further understanding of the test score. The fact that the 
five correct items are not all ones that were easy for the class as a whole 
was the reason (but not the explanation) of the high MCI. Two other students 
did not make the cut-off of .30, but have high MCIs: the first, with .29, and 
the next to last, with .26. In general, any item missed above the S-curve is 

unusual since it is relativelv eA<:v f nr - i-v,« i / 

eiy easv for tne class (see student one and item 

14). On examination it may appear to be as simple an explanation as 
carelessness. But, too, thes. items might represent missed class days, and/or 
concepts unlearned. 

When items are analyzed by objectives (or content domain), teachers may 
obtain information on a student's understanding of that concept. Specific 
lists of which objectives have been mastered by individual students are given 
by TAP (Appendix 7). SPP can print an S-P Chart organized by objective 
(Appendix 11), which gives mo re detailed information. From a Categorized S-P 
Chart the teacher can see which objectives are troublesome to which students, 



and which items within each objective are missed. In the example chart in 
Appendix 11, the high MCI for the last student is clearly explained. Almost 
all of the items he got correct were factoring problems. Here the teacher has 
uncovered an unexpected strength. It is possible the student had extra 
tutoring in this area, or just vnderstood it very well. The Categorized S-P 
Chart can pinpoint problem areas for concentrated work, and find students with 
strengths who can be paired v c 1 those who need help. 

Item Analysis 

It is important to a teacher who wishes to bui^ ; a bank of test 
questions to determine which items are appropriate for using again. Items 
with incorrect answer keys, or ones that might not have been covered 
completely in the classroom may also be identified. SPP reports the items in 
order of difficulty (Appendix 9), including the percentage of students getting 
the item correct. Notice that item 3 was very easy for this class (all got it 
correct), while item 10 was very difficult (only one student got it correct). 
The listing of item MCIs and MCSs (Appendix 9) gives an indication of unusual 
patterns of responses. Response patterns for items are unusual to the extent 
that items are answered incorrectly by high scoring students (i.e., high total 
score) and correctly by low scoring studen:s. Thir situation suggests random 
responses by students (guessing), or something unusual about the content of an 
item, e.g., its wording or the nature of what is being asked. The MCI of an 
item is an indicator of item fit to the classroom of students. If the item 
MCI is greater than .30 (that is, indicated with 'X' or 'Z' for its MCS), the 
item should be reviewed before it is used again. Note that three items have 
been identified in this manner: 14, 16, and 7. 
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The study of which students -ended to get an item correct is also usetul 
in item analysis. The point-biserial correlation between an item (0/1) and 
the total test score can be an indication of this type. A high point-bise~ial 
me - . that the students with high s, ,s tend to get the item correct, wnile 
students with low scores tend to get it wrong. The TAP report of point- 
bis,rials can be seen in Appendix 3. Note that item 1 has a high value for 
the fourth response option (the correct answer, indicated with an asterisk). 
This implies that the item is behaving properly in this classroom. Note that 
the point-biserial for item lo is rather low, and that two of the distractors 
have positive correlations too. More evidence that this item suould be 
studied. Similar information can be found by lucking at the percent of 
correct responses by quJntiles (Appendix 4). It would be expected that the 
5th quintile should get the highest percent correct, gradually decreasing down 
to the 1st quintile. Notice that item 7 does not follow this pattern at all. 
Although 50% of the 5th quintile got item 7 correct, it drops to zero for the 
4th and 3rd. then back up to 25% ... 33% for the 2nd and 1st respectfully. 
This pattern can be seen graphically in Appendix 5. This item is performing 
quite curiously in this classroom. 

In performing an item analysis, the distractors (the incorrect options) 
should be carefully examined. A distractor should not attract ,ery good 
students (a sign of a tricky question or incorrect answer key). A good 
distractor should tell tne teacher what misvnderstanding has caused the 
student to answer the item incorrectly. By looking at the responses chosen by 
quintiles (Appendix 6), a teacher can determine which distractors are favored 
by .ach gr.ap. Note that for item 7, the higher quintiles are split between 
option 1 and 5 (4 is the correct answer), but the lower groups tend to choose 
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option 5. S-P Charts (Appendix 8) can give similar information about 
distractors. By looking down the column of an item one can see if the higher 
scoring students (those at the top of the chart) are choosing different 
distractors than those nearer the bottom. Item 6 is a good example for this 
classroom. Note that the higher scoring students are choosing 3, while the 
low scorers are tending to choose 4. This type of analysis can aid in the 
understanding of right/wrong answers as well as decisions on whether an item 
should be included in a test bank. 

Classroom Analysis 

TAP and SPP reports present much of the basic classroom information a 
teacher may need. To get an idea of general test performance, the mean test 
score and the distribution of test scores can be useful, as well as the 
reliability (Append* 2). The disparity coefficient is an index of how well 
the test fits this classroom of students (Appendix 9). For this classroom, 
the disparity coefficient is .44, well below the caution point of .60, 
signifying a relatively good test for this class. More than likely all the 
information tested was presented in some form to these students. 

By grouping items into objectives or content domains, the information 
from SPP and TAP can be extremely useful in guiding instruction, since they 
contains important summary information for use by the classroom teacher. One 
can identify the content areas on which the classroom as a whole might need 
further instruction and practice. By grouping items in this manner, the 
information more closely matches the way teachers presents topics in the 
classroom. TAP reports the percent cf students mastering each objective, as 
well as the reliability of the subtest created by the objective (Appendix 7b). 
In this sample report, 95% of the class mastered objective 3 (polynomials), 
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with its average item difficulty of .84 (on the average, 84% of the students 
got each item in this objective correct). Using a Categorized S-P Chart the 
results of a test can be utilized quickly. In fact, many teachers find the 
Categorized S-P Charts quite useful (Switzer, 1987). The teacher can readily 
see which content areas are understood by mu ch of the class (taose listed on 
the left side of the chart) and therefore can be considered their strengths. 
The last few topics may need extra review and concentration. Since items are 
ordered easiest to hardest within each category, this helps the teacher spot 
those items within each objective that are giving the classroom the most 
trouble. 

Summary 

Analysis of item responses using TAP and SPP reports provide a method of 
systematically looking at student achievement patterns on common areas of 
content. Detailed output provides a teacher with concise reports identifying 
instructional areas in need of review, students with unusual response pattern, 
and items that need further study. This paper has attempted to demonstrate 
the potential usefulness of such reports for classroom teachers. 
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APPENDIX 1A 
TAP and SPP Data and Key Files 

Data File: 

9218354113424331353525441211 
9205304113425334353324441211 
9200103113421143353555231211 
9215455113535333453255351213 
92^5504513435132253355341221 
9202954513431342353555351211 
9210954212445143433445351211 
9209752413545313451342441211 



Answer ke^' c'ile: 
4113424331353324441211 



APPENDIX IB 
TAP: Scored Responses to Items 
STUDENT RESPONSES TO TEST ITEMS 
STUDENT 

ID NO. ITEM NO. 

0000000001111111111222 
1234567890123456789012 

921835 1111111111111010111111 

920530 1111110110111111111111 

920010 0111110000111000001111 

921545 0111000110011000001110 

920550 1011100010011100011101 

920295 1011100100111000001111 

921095 1010100000001000001111 

920975 0011000100010100111111 



1 - CORRECT RESPONSE 
0 - INCORRECT RESPONSE 
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APPENDIX 2 
TAP: Descriptive Statistics 

STUDENT SUMMARY 



NUMBER OF TEST ITEMS 


- 


22 




AVG. TEST SCORE 




12 


25 


STD. DEV. OF TEST SCORE 




3 


92 


STD. ERROR OF MEAN 


: 


0 


88 


SKEWNESS 




0 


57 


KURTOSIS " 


cm 


2 


73 


NUMBER OF TEST SCORES 




20 


ITEM SUMMARY 








AVG. ITEM DIFFICULTY 




0 


56 


STD. DEV. OF 








ITEM DIFFICULTY 




0. 


28 


TEST SUMMARY 








RELIABILITY COEF. (KR-20) 




0. 


79 



Frequency of Total Test Scores 

FREQUENCY DISTRIBUTION OF TEST SCORES 

EACH * 

RAW STANDARDIZED CUM REPRESENTS 

SCORE SCORE FREQ FREQ 1 PERSON 



20 


698 


2 


20 


** 


19 


672 


1 


18 


* 


18 


647 


0 


17 




17 


621 


0 


17 




16 


596 


0 


17 




15 


570 


1 


17 


* 


14 


5*5 


2 


16 


** 


13 


519 


0 


14 




12 


494 


5 


14 


***** 


11 


468 


i 


9 


** 


10 


443 


4 


7 


**** 


9 


417 


0 


3 




8 


392 


2 


3 


** 


7 


366 


0 


1 




6 


341 


0 


1 




5 


315 


1 


1 


* 
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TAP: Point Biserial Correl 

POINT-BISERIAL CORRELATIONS BETWEEN 
RESPONSE OPTIONS AND TOTAL 
TEST SCORE 

ITEM RESPONSE OPTION 

#12 3 4 5 OMIT 

1 -0.41 -0.21 -0.02 0.53*-0.14 0 00 

2 0.64**0.19 -0.17 -0.37 -0.03 -0.14 

3 0.00* 0.00 0.00 0.00 0.00 0.00 

4 -0.44 -0.20 0.43* 0.00 0.00 0 00 

5 0.00 -0.11 -0.44 0.38*-0.13 0.00 

6 0.00 0.47* 0.16 -0.62 0.C0 0.00 

7 0.08 0.00 -0.44 0.26*-0.09 0 00 

8 0.53 0.00 0.51* 0.17 -0.15 000 

9 -0.15 0.11 0.34*-0.38 0.17 0 00 

10 0.47*-0.03 -0.41 0.29 -0.02 0 00 

11 -0.17 -0.10 0.41**0.19 0.00 -0.02 

12 -0.24 -0.44 -0.26 -0.14 0.58* 0.00 

13 -0.08 0.00 0.37* 0.00 -0.44 0 00 

14 0.00 -0.20 0.25**0.26 -0.02 0 00 

15 -0.28 0.60* 0.00 -0.47 -0.10 0 00 

16 0.11 -0.37 -0.20 0.26* 0.07 0 00 

17 0.02 -0.10 -0.19 0.45*-0.44 0 00 

18 -0.08 0.00 -0.33 0.59*-0.30 0 00 

19 0.41*-0.14 -0.44 0.00 0.00 o!oO 

20 -0.44 0.44* 0.00 0.00 0.00 0 00 

21 0.43*-0.43 0.00 0.00 0.00 0 00 

22 0.32* 0.00 -0.32 0.00 0.00 0.00 

* INDICATES CORRECT RESPONSE OPTION 
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APPENDIX 4 

TAP: Analysis by Quintiles 

QUINTILE FREQUENCY DISTRIBUTION 
OF TEST SCORES 

NO. OF PROPORTION SCORE 
QUINTILE SCORES OF SCORES RANGE 



1ST 3 0.15 5 - 9 

2ND 4 0.20 10 - 10 

3RD 2 0.10 11 - 11 

4TH 7 0.35 12-14 

5TH 4 0.20 15 - 20 



PERCENT OF CORRECT RESPONSES 
TO TEST ITEMS BY QUINTILES 



ITEM # 


1ST 


2ND 


3RD 


4TH 


5TH 


1 


33 


0 


0 


43 


100 


2 


0 


25 


0 


14 


100 


3 


100 


100 


100 


100 


100 


4 


33 


100 


100 


86 


100 


5 


67 


25 


50 


71 


100 


6 


0 


25 


0 


43 


75 


7 


33 


25 


0 


0 


50 


8 


0 


50 


50 


71 


75 


9 


"!3 


25 


50 


43 


75 


10 


0 


0 


0 


0 


25 


11 


33 


0 


0 


29 


75 


12 


0 


75 


50 


100 


100 


13 


67 


100 


50 


100 


100 


14 


33 


25 


50 


57 


75 


15 


0 


25 


50 


43 


100 


16 


0 


25 


50 


14 


50 


17 


33 


25 


50 


14 


75 


18 


0 


25 


50 


43 


75 


19 


67 


75 


100 


100 


100 


20 


67 


100 


100 


100 


100 


21 


33 


100 


100 


86 


100 


22 


67 


50 


100 


100 


100 
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APPENDIX 5 

TAP: Graphical Representatioi. of Percent Correct by Quintile 
Percent Of Correct Responses By Quintile For Item Number 7 



Q 
U 

I 
N 
T 
I 
L 
E 



1ST 



2ND 



3RD 



4TH 



5TH 



***************** 
**************** 

************* 
************ 



************************* 
*********** ************** 



-+ + + + + + , 

1 2 3 4 5 6 

oooooo 

PERCENT CORRECT 



■+- 
7 
0 



-+- 
8 
0 



-+- 
9 
0 



Percent Of Correct Responses By Quintile For Item Nun-ber 18 
1ST 



Q 
u 

I 

N 
T 
I 
L 
E 



2ND 



3RD 



4TH 



5TH 



************* 
************ 

************************* 
************************* 

********************** 
********************* 

******************************** ieic . kieieil 
******************************** ieicicieie 



-+ + + + + + + 

1 2 3 4 5 6 7 

ooooooo 

PERCENT CORRECT 



-+- 
8 
0 



-+- 
9 
0 
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APPENDIX 6 

TAP: Analysis of Responses by Quintiles 



*** ITEM 7 *** 

FREQUENCY OF RESPONSES BY QUINTILES 

RESPONSE QUINTILE PROP. OF 

OPTION 1ST 2ND 3RD 4TH 5TH RESPONSE 



1 


0 


0 


1 


3 


1 


0 


25 


2 


0 


0 


0 


0 


0 


0 


00 


3 


1 


0 


0 


0 


0 


0 


05 


4* 


1 


1 


0 


0 


2 


0 


20 


5 


1 


3 


1 


4 


1 


0. 


50 


OMIT 


0 


0 


0 


0 


0 


0. 


00 



* INDICATES CORRECT RESPONSE 



*** ITEM 18 *** 
FREQUENCY OF RESPONSES BY QUINTILES 
RESPONSE QUINTILE PROP. OF 



OPTION 


1ST 


2ND 


3RD 


4TH 


5TH 


RESPON! 


1 


0 


0 


1 


0 


0 


0.05 


2 


0 


0 


0 


0 


0 


0.00 


3 


1 


1 


0 


2 


0 


0.20 


4* 


0 


1 


1 


3 


3 


0.40 


5 


2 


2 


0 


2 


1 


0.35 


OMIT 


0 


0 


0 


0 


0 


0.00 



* INDICATES CORRECT RESPONSE 
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APPENDIX 7 
TAP: Objectives Mastered 



TOTAL (% OF) 
STUDENT SKILLS 



ID 



MASTERED OBJECTIVE SCORES (* - MASTERY) 







1 


2 


3 


4 




a / 1 r * a a \ 
4 ( 1UU . U) 


7* 


3* 


7* 


3* 


7< UJ JU 


A / 1 A A AN 

4(100. 0) 


7* 


2* 


7* 


4* 




1( 25.0) 


4 


0 


6* 


2 




1( 25.0) 


2 


1 


6* 


1 


920550 


0 ( SO C\\ 




l 


6* 


3* 


920295 


1( 25.0) 


2 


1 


7* 


2 


921095 


1( 25.0) 


1 


0 


6* 


1 


920975 


2( 50.0) 


2 


2* 


5* 


2 


920685 


1( 25.0) 


0 


0 


1 


4* 


920485 


1( 25.0) 


3 


1 


7* 


1 


920645 


1( 25.0) 


2 


1 


6* 


1 


920825 


2( 50.0) 


3 


1 


5* 


3* 


920760 


1( 25.0) 


4 


1 


5* 


0 


920600 


1( 25.0) 


3 


0 


6* 


2 


921685 


2( 50.0) 


4 


2* 


6* 


2 


920230 


1( 25.0) 


3 


0 


5* 


0 


$20970 


3( 75.0) 


3 


2* 


6* 


3* 


921385 


1( 25.0) 


2 


0 


6* 


2 


920130 


2( 50.0) 


6* 


0 


7* 


2 


920398 


4(100.0) 


6* 


2* 


7* 


4* 



OBJECTIVE LEGEND 

1- Equations 

2- Inequalities 

3- Polynomials 

4- Factoring 



AVG. 

OBJECTIVE OBJ. 
NUMBER SCORE 



1 

2 
3 
4 



3.30 
1.00 
5.85 
2.10 



STD. 
DEV. 



STD. 
ERROR 
OF 
MEAN 



SKEW- 
NESS 



1.92 0.43 0.57 

0.92 0.21 0.39 

1.35 0.30 -2.19 

1.21 0.27 -0.01 



APPENDIX 7B 
TAP: Objective Summaries 
# OF % OF 



KUR- 
TOSIS 

2.36 
1.97 
8.56 
2.02 



STU- 


STU- 


AVG. 


STD. 


RELIA- 


DENTS 


DENTS 


ITEM 


DEV. OF 


BILITY 


MAS- 


MAS- 


DIFFI- 


ITEM 


COEF. 


TERING 


TERING 


CULTY 


DIFF. 


(KR-20) 


4 


20.. 


0.41 


0.19 


0.62 


6 


30.0 


0.33 


0.26 


0.55 


19 


95.0 


0.84 


0.20 


0.70 


7 


35.0 


0.52 


0.23 


0.57 



ERIC 



23 



APPENDIX 8 



SPP: The S-P Chart 

Student Problem (S-P) chart analysis of Basic Algebra 
School: Eastside Instructor: D. Smith (#92) 

Number of students: 20 Number of problems: 22 

Problem Number 

Modified 



Student 


Test 


Score 


Caution 


o 


2 


1 


1 


0 9 


0 


1 0 
x u 


v X 


0 


1 


0 


1 


0 


1 


U 


1 


1 0 


1 


Number 


(Rav)< % ) 


TnH /C an 


j 


n 

u 


J 


Q 


A 1 


o 
I 


0 c 

1 D 


8 4 


9 


5 


1 


8 


6 


7 


2 


1 


6 7 


0 


Answer Key - 






i 
i 


o 
L 


•J 
J 


1 

L 


3 1 


L 


5 4 


3 3 


3 


2 4 4 


2 4 


L 


3 4 4 


L 


921835 


20 


90.9 


O 0 0 A 
U . Z7 A 


+ 


+ 


+ 


+ 


+ + 


+ 


+ + 


+ 5 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


5S+ 


+ | 


920530 


20 


90.9 


0.00 A 


j + 


+ 


+ 


+ 


+ + 


+ 


+ + 


+ + 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


+S5 


pj 
4| 


920398 


19 


86.4 


0.06 A 


j + 


+ 


+ 




+ + 


+ 


+ + 


+ + 


+ 


+ 


+ 


+ 


3 


+ 


+ 


+S5 + 


4| 


920130 


15 


68.2 


0.11 A 


j + 


+ 


+ 


+ 


+ + 


+ 


+ + 


4 + 


5 


+ 


+ 


5S-;- 


3 


+ 


4 


+ i 


3| 


921685 


14 


63.6 


0.10 A 


j + 


+ 


+ 


+ 


+ + 


+ 


+ 5 


+ + 


2 


+ 


2S+ 


3 


+ 


2 


4 


P 

r 

•«■ 1 


4| 


920970 


14 


63.6 


0.02 A 


j + 


+ 


+ 


+ 


+ + 


+ 


+ + 


+ + 


+ 


5 


2S+ 


+ 


1 


3 


1 


F 

1 5 


2| 


920010 




































P 


P 






12 


54.5 


0.11 A 


j + 


+ 


+ 


+ 


+ + 


+ 


+ + 


1 5 


4S5 


3 


3 


+ 


2 


+ 


+ 


5 1 


3| 


































P 


p 










920550 


12 


54.3 


0.12 A 


j + 


+ 


+ 


+ 


+ 2 


+ 


+ + 


1 + 


+S5 


+ 


+ 


3 


3 


5 


2 


5 5 


2| 






























P 


P 














920295 


12 


54.5 


U . \JD A 


1 + 


+ 


+ 


+ 


+ + 


+ 


+ + 


+ 5 


4S5 


+ 


5 


3 


3 


5 


+ 


5 1 


2| 


























P 


P 


















920485 


12 


54.5 




1 + 


+ 


+ 


4- 


+ + 


+ 


+ 2 


+ 5 


4S+ 


+ 


5 


+ 


3 


5 




5 5 


3| 


920825 


12 


54.5 


0.08 A 


| + 


+ 


+ 


+ 


2 + 


+ 


+ + 


P 

+ + 


+S+ 


3 


3 


3 


2 


2 


2 


5 5 


5| 


920975 


11 


50.0 


0.15 C 


+ 


+ 


1 


+ 


+ + 


+ 


+ 5 


+ +S1 


4 


2 


+ 


4 


+ 


4 


4 


2 5 


3| 


920600 


11 


50.0 


0.12 C 


+ 


+ 


+ 


+ 


+ + 


+ 


1 + 


5 5S+ 


+ 


2 


1 


4 


1 


2 


4 


+ 1 


3| 


921545 


10 


45.5 


0.14 C 


| + 


+ 


+ 


+ 


+ + 


3 


P 

+ 5 


+S2 


+ 


5 


5 


5 


3 


3 


+ 


4 


5 5 


3| 


920645 


10 


45.5 


0.19 C 


| + 


+ 


+ 


+ 


+ + 


+ 


4 5 


5S2 


4 


1 


1 


+ 


4 


+ 




2 


3 + 


2| 


920760 


10 


45.5 


0.20 C 


+ 


+ 


+ 


2 


+ + 


3 


P 

+ + 


+S5 


1 


+ 


2 


5 


4 


3 


2 


1 


+ 5 


3| 


921385 


10 


45.5 


0.07 C 


+ 


+ 


+ 


+ 


+ + 


+ 


+ 2 


1S+ 


4 


5 


2 


3 


+ 


2 


3 


4 


3 5 


3| 


















? P 


P 


























921095 


8 


36.4 


0.11 C 


+ 


+ 


+ 


+ 


2 + 


+ 


3S+ 


1 4 


4 


4 


+ 


5 


4 


3 


2 


4 


5 5 


3| 














P 


P 






























920230 


8 


36.4 


0.26 C 


+ 


+ 


+ 


+ 


+ 2 


3 


1S+ 


1 5 


4 


1 


2 


5 


4 


+ 


3 


1 


5 -t 


3| 


920685 


5 


22.7 


0.43 D 


+ 


P 
1 


5 


3 


1S2 


+ 


2 3 


1 + 


+ 


4 


1 


3 


4 


5 


4 


+ 


2 3 


41 



9 
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APPENDIX 9 

SPP: Descriptive Statist! 

**** PROBLEM STATISTICS **** 
Permuted a 
PrnK1 „„ . „ . , Modified 

Problem Answer Problem Percent Caution 
Number Key Total Correct Ind/Sgn 



cs 



3 
20 
13 
19 
4 
21 
22 
12 
5 
8 
14 
9 
15 
1 
18 
6 
17 
2 
11 
16 
7 
10 



1 

2 
3 
1 
3 
1 
1 
5 
4 
3 
3 

3 

2 

4 

4 

2 

4 

1 

3 

4 

4 

1 



20 
19 
18 
18 
17 
17 
17 
15 
13 
11 
10 
9 
9 
8 
8 
7 
7 
6 
6 
5 
4 
1 



100.0 
95.0 
90.0 
90.0 
85.0 
85.0 
85.0 
75.0 
65.0 
55.0 
50.0 
45.0 
45.0 
40.0 
40.0 
35.0 
35.0 
30.0 
30.0 
25.0 
20.0 
5.0 



0.00 

0.00 

0.11 

0.07 

0.11 

0.11 

0.18 

0.02 

0.21 

0.15 

0.33 

0.27 

0.09 

0.15 

0.11 

0.21 

0.23 

0.12 

0.27 

0.38 

0.40 

0.00 



**** STUDENT SUMMARY **** 

Average raw score _ 12 25 

Standard deviation of raw score - 3 '92 

Average percent of items correct - 55*68 

Average modified caution index - 0*13 
Standard deviation of modified 

caution index - 0 10 

**** PROBLEM SUMMARY **** 
Average item difficulty (%) _ 55 68 

Standard deviation of item 

difficulty - 5 63 

Average modified caution index - 0 16 
Standard deviation of modified 

caution index 



- 0.12 



**** TEST SUMMARY 
Average overall student 

performance on test (%) 
Reliability coefficient 

(Cronbach's alpha) 
Disparity coefficient 



**** 



Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

X 

W 

W 

w 
w 
w 
w 
w 
w 

X 
X 

w 



55.68 

0.79 
0.44 



APPENDIX 10 



SPP: Charts for Students and Items 



**** CLASSIFICATION AND FREQUENCY **•* 
**** DISTRIBUTION TABLES **** 



(1) Number of students in each cell classified by 
test performance and mooified caution index 

Student Student Percent Modified 
Performance Cell Correct Caution Index 

+ -» + 100% 

I A I B | A > 50% <- 0.30 

HIGH | | | 

I 11 I 0 I B > 50% > 0.30 

+ + + 50% 

I c I D I C <- 50% <- 0.30 

LOW | | | 

I 8 I 1 I D <- 50% > 0.30 

+ + + 0% 

0 0.30 
Modified Caution Index 



(2) Number of items in each cell classified jy item 
difficulty and modified caution index 

Item Item Percent Modified 
Difficulty Cell Correct Caution Ir 
+ + + 0% 

I w I X | W <- 50% <- 0.30 

HARD | | | 

I 9 I 3 I X <- 50% > 0.30 

+ + + 50% 

I Y | Z | Y > 50% <- 0.30 

EASY | | I 

I 1° I 0 I Z > 50% > 0.30 

+ + + 100% 

0 0.30 
Modified Caution Index 



APPENDIX 11 



SPP: The S-P Chart of Objectives 
Student Problem (S-P) chart analysis of Basic Algebi 



>ra 



School: Eastside 
dumber of students; 



20 



Instructor: D. Smith (#92) 
Number of problems: 22 



Problem Number 

Modified 

Student Test Score Caution 0211020 2101 10101010 011 
Number (Raw)( % ) Ind/Sgn 3039411 2491 25567267 880 



Category Name 
Answer Key — 



921835 


20 


90 


.9 


0 


.29 A | 


920530 


20 


90 


.9 


0 


.00 A | 


920398 


19 


86 


.4 


0 


.06 A | 


920130 


15 


68 


.2 


0 


.11 A | 


921685 


14 


63 


.6 


0 


.10 A | 


920970 


14 


63 


.6 


0 


.02 A | 


920010 


12 


54 


.5 


0 


.11 A | 


920550 


12 


54 


.5 


0 


.12 A | 


9^0295 


12 


54 


.5 


0 


05 A | 


920485 


12 


54 


5 


0 


08 A | 


920825 


12 


54 


5 


0. 


08 A | 


920975 


11 


50. 


0 


0. 


15 C | 


920600 


11 


50. 


0 


0. 


12 C | 


921545 


10 


k5. 


5 


0. 


14 C | 


920645 


10 


45. 


5 


0. 


19 C | 


92076C 


10 


45. 


5 


0. 


20 C | 


921385 


10 


45. 


5 


0. 


07 C | 


921095 


8 


36. 


4 


0. 


11 C | 


920230 


8 


36. 


4 


0. 


26 C j 


9206*5 


5 


22. 


7 


0. 


43 D | 



-> Polyncm Fctr Equation Ineq 
-> 1231314 1333 54224144 341 



++H l il t 
+4 I M M 



+5h+ 



■ H M 11 + 
+++++++ 
+- H I 1 f 2 
M 1 I I f 2 
++++++3 

+++++2+ 
f I I I I I I 
■ H I M + + 



++1+++2 

++++++2 
HI M f 3 

++++K1 
+++2. >1 
++++++2 
++++2++ 
+++++22 
+^53121 



++++ 

++54 

++24 

+++1 

+54+ 

+++2 

+54+ 

+54 

+++2 

++14 

+5+4 

32+4 

+242 

3511 

++44 

+444 

3541 



++++++5+ +++ 
- Hill 1+ 5 ++4 
+++3++S+ ++4 
++++3++1 453 
+5+3+2+1 ++4 
++5+131.5 -f+2 
H-5+2+51 133 
++533555 1+2 
++533551 +52 
+2++355S +53 
+++32255 +35 
+544+425 -3 
1++412+1 ;a3 
+5533+55 +53 
4514+ 3+ 5+2 
+++432+5 +53 
+25+2335 133 
3+443255 153 
1+14+35+ 153 
23445423 134 



Category Category Percent 
Rank Name 



Correct 



1 Polynomials 83 

* Fac Coring 52 

3 Equations 41 

4 Inequalities 33 



27 



